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Abstract 


Different  formulations  and  measures  that  may  be  used  for  evaluating  the  complexity  of  systems 
are  gathered  in  this  Technical  Note.  They  might  be  useful  for  describing  aspects  of  military 
complex  systems.  They  were  extracted  from  documents  issued  from  the  scientific  literature 
dedicated  to  Complexity  Theory,  chaos  and  complex  systems. 


Resume 


Differentes  formulations  et  mesures  qui  peuvent  etre  utilisees  pour  evaluer  la  complexite  des 
systemes  sont  rassemblees  dans  ce  document.  Ces  formulations  et  mesures  peuvent  etre  utiles 
pour  decrire  differents  aspects  des  systemes  militaires  complexes.  Elies  ont  ete  extraites  de 
documents  provenant  de  la  litterature  scientifique  traitant  de  la  theorie  de  la  complexite,  du  chaos 
et  des  systemes  complexes. 
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1  Introduction 


This  document  contains  the  results  of  numerous  searches  that  were  made  in  the  context  of  a  state- 
of-the-art  in  the  domain  of  the  Complexity  Theory,  chaos  and  complex  systems.  It  aims  at 
regrouping  together  current  ways  of  formulating  and  measuring  complexity  of  complex  systems. 

1.1  Context  and  Scope  of  this  Document 

This  document  is  the  second  of  a  set  of  five  DRDC  Valcartier  reports  dedicated  to  the  study  of 
complexity  theory,  chaos  and  complex  systems  (Couture,  2006a,  2006b,  2006c,  2006d,  and  one  to 
be  published  in  2007).  It  is  part  of  an  overarching  project  being  carried  on  at  DRDC  Valcartier, 
Project  15bp01  -  Defensive  Software  Design.  It  focuses  mainly  on  the  presentation  of  concepts 
from  this  theory.  There  are  only  a  few  references  to  the  architecting  and  engineering  aspects  of 
complexity.  These  aspects  will  be  covered  in  another  document. 

1.2  The  Used  Methodology 

Figure  1  depicts  the  general  methodology  used  for  this  study.  It  is  characterized  by  a  main 
iterative  and  incremental  loop  (steps  1  and  2),  which  includes  a  number  of  sequential  and  parallel 
activities  (steps  3,  4  and  5).  This  loop  permits  on-the-fly  adjustment  and  optimization. 

The  five  main  activities  or  steps  are: 

1.  Search  literature,  projects,  groups,  etc:  Internet  searches  were  made  using  Google  and 
other  search  engines.  A  number  of  specialized  databases  were  also  searched  (Dialog  Database 
Catalog,  2005).  These  databases  are  listed  in  Annex  C  of  Couture  (2006a). 

2.  Select  potentially  useful  documents:  Documents  were  selected  based  on  their  potential 
applicability  to  the  military  context. 

3.  Study  selected  documents:  Approximately  30-40%  of  the  selected  documents  were  read  and 
studied  in  greater  detail. 

4.  Investigate  in  greater  depth  for  the  military  context:  This  involved  finding  elements  that 
offer  potential  solutions  in  the  military  context. 

5.  Update  documents:  The  content  of  each  document  was  updated  on  the  fly  at  each  iteration. 


DRDC  Valcartier  TN  2006-451 


1 


Update  Documents 


5) 


Update 


Literature 

Review 

Documents 


5) 


-Update 


Concept  Review 
Document 


5) 


Update 


Solution 

Documents 


Figure  1  The  Methodology  Used  in  this  Work. 

The  reports  generated  by  this  study  are  listed  in  Table  1.  The  first  four  reports  will  be  published 
by  the  end  of  phase  one  of  the  study  (by  March  2007).  The  last  will  be  published  by  the  end  of 
phase  two  (late  2007). 
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Table  1  List  of  Documents  Issued  from  this  Work. 


Title 

Description 

Complexity  and  chaos  -  State-of- 
the-art;  List  of  works,  experts, 
organizations,  projects,  journals, 
conferences  and  tools. 

(Couture,  2006a). 

This  Technical  Note  provides  471  references  to  scientific 
studies,  organizations,  scientific  journals,  conferences, 
experts  and  tools,  plus  713  additional  Internet  addresses 
that  are  related  to  complexity  theory,  chaos  and  complex 
systems.  Abstracts  are  included  where  available. 

Complexity  and  chaos  -  State-of- 
the-art;  Formulations  and 

measures  of  complexity. 

(This  document). 

Different  formulations  and  measures  of  system  complexity 
are  provided  in  this  Technical  Note.  They  were  drawn 
from  the  scientific  literature  on  complexity  theory,  chaos 
and  complex  systems. 

Complexity  and  chaos  -  State-of- 
the-art;  Glossary. 

(Couture,  2006c). 

This  Technical  Note  defines  335  key  words  related  to 
complexity  theory,  chaos  and  complex  systems.  The 
definitions  were  extracted  from  the  scientific  literature. 

Complexity  and  chaos  -  State-of- 
the-art;  Overview  of  theoretical 
concepts. 

(Couture,  2007). 

This  Technical  Memorandum  presents  an  overview  of 
theoretical  concepts  pertaining  to  complexity  theory. 

Complexity  and  chaos  -  State-of- 
the-art;  The  Engineering  of 
complex  adaptive  systems. 

(To  be  published  in  2007). 

Descriptions  of  the  current  approaches,  methodologies  and 
tools  used  to  address  problems  related  to  the  architecting, 
engineering  and  improvement  of  complex  systems  is 
included  in  this  Technical  Report. 

1.3  How  to  use  this  Document 

This  report  is  a  starting  point  reference  for  the  search  of  information  related  to  the  current  ways  to 
formulate  and  measure  the  complexity  of  systems.  Considering  the  quantity  of  information 
included  in  this  document,  three  ways  of  searching  are  proposed: 

•  To  get  a  general  survey  on  a  specific  subject:  The  reader  may  simply  use  the  Table  of 
Content  for  locating  specific  subjects. 

•  To  find  specific  information  on  a  specific  subject:  The  reader  must  use  its  word 
processor’s  search  engine  to  find  all  appearances  of  selected  key  words  within  this 


DRDC  Valcartier  TN  2006-451 


3 


document.  S/he  will  also  find  appropriate  key  words  and  addresses  that  can  be  re-used  in 
further  searches  over  the  Internet  for  instance. 

Edmonds’  list  of  references  is  also  an  important  source  of  information  (Edmonds,  1999). 
The  reader  is  invited  to  refer  to  this  important  PhD  Thesis. 
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2  Formulations  and  Measures  of  Complexity 


This  Chapter  contains  a  number  of  64  formulations/measures  of  complexity  for  complex  systems 
pertaining  to  different  domains  or  disciplines.  A  copy  of  Edmonds’  list  of  500  references 
(Edmonds,  1999)  has  been  included  in  Annex  A  in  order  to  facilitate  the  finding  of  cited 
documents. 

2.1  Abstract  Computational  Complexity 

(Edmonds,  1999):  Blum  [67]  proposed  an  abstract  definition  of  computational  complexity.  If 
pi(n)  are  the  functions  representing  the  computation  of  the  programs  Pi,  then  ci(n)  are  a  set  of 
complexity  measures  iff  ci(n)  is  defined  exactly  when  pi(n)  is  defined  and  the  predicate  ci(n)=m  is 
decidable.  This  definition  neatly  includes  the  time  and  space  measures  as  well  as  many  other 
sensible  resource  measures  (such  as  the  number  of  jumps  executed)  and  is  strong  enough  to  prove 
many  of  the  important  theories  concerning  them.  However  this  definition  is  too  broad  as  it  allows 
measures  which  don't  obey  the  subprogram  property  (if  P  is  a  program  that  first  applies  a 
subprogram  Q  to  an  input  and  then  a  subprogram  R  to  the  result,  then  the  complexity  of  P  should 
be  at  least  as  great  as  that  of  Q  or  R).  Thus  according  to  this  approach  you  get  programs  with 
more  complex  subprograms.  Fixes  for  the  abstract  definition  of  computational  complexity  are 
suggested  by  Turney  in  [444,  445,  446],  and  Ausiello  suggests  a  weakened  version  in  [31].  [66] 
argues  that  computational  complexity  should  be  extended  over  other  fields  like  the  real  numbers. 

2.2  Algorithmic  Complexity 

(Flake,  1998):  The  size  of  the  smallest  program  that  can  produce  a  particular  sequence  of 
numbers.  Regular  patterns  have  low  algorithmic  complexity  and  random  sequences  have  high 
algorithmic  complexity. 

2.3  Algorithmic  Information  Complexity 

(Edmonds,  1999):  The  Algorithmic  Information  Complexity  (AIC)  of  a  string  of  symbols  is  the 
length  of  the  shortest  program  to  produce  it  as  an  output.  The  program  is  usually  taken  as 
running  on  a  Turing  Machine.  It  was  invented  by  Solomonoff  [419] ,  Kolmogorov  [266]  and 
Chaitin  [99,  100,  101]  separately,  although  perhaps  anticipated  by  Vitushkin  (section  8.45  on 
page  161).  It  has  been  one  of  the  the  most  influential  complexity  measures  (along  with  that  of 
computational  complexity)  and  has  inspired  many  variations  and  enhancements  including 
‘sophistication’  (section  8.42  on  page  159),  and  ‘logical  depth’  (section  8.4  on  page  138). 
Although  Solomonoff  considered  it  as  a  candidate  for  selection  amongst  equally  supported 
scientific  theories  (i.e.  a  measure  of  simplicity  -  section  8.37  on  page  156),  Kolmogorov  and 
Chaitin  considered  it  as  a  measure  of  information  (see  section  8.15  on  page  144  and  section  8.24 
on  page  149).  It  has  many  interesting  formal  properties  [99],  including:  1.  The  more  ordered  the 
string,  the  shorter  the  program,  and  hence  less  complex.  2.  Incompressible  strings  (those  whose 
programs  are  not  shorter  than  themselves)  are  indistinguishable  from  random  strings.  3.  Most 
long  strings  are  incompressible.  4.  In  a  range  of  formal  systems  you  can't  prove  (within  that 
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system )  that  there  are  strings  above  a  certain  fixed  level  of  complexity  (derived  basically  from  the 
AIC  of  its  axioms).  5.  In  general  it  is  uncomputable.  Property  2  illustrates  the  deep  connection 
between  AIC  and  disorder.  This  is  particularly  evident  in  physics  where  a  very  close  connection 
between  Algorithmic  complexity  and  entropy  has  been  shown  [449] ,  to  the  extent  that  it  is  often 
referred  to  as  an  entropy.  Property  4  indicates  that  the  AIC  complexity  is  more  of  an  information 
measure.  While  one  might  believe  that  it  is  not  possible  to  produce  more  information  within  a 
formal  system  than  is  encoded  by  the  axioms,  it  would  be  extremely  counter-intuitive  if  there  was 
a  limit  to  how  complex  one  could  prove  strings  in  it  to  be.  AIC  has  been  applied  in  many  ways:  to 
define  randomness  in  a  non-probabilistic  way  [309,  499] ;  to  capture  descriptive  complexity 
[293]  (see  also  section  8.8  on  page  141);  Rissanen  uses  a  statistical  version  to  motivate  a 
principled  trade-off  between  the  size  of  model  and  its  error  in  [377,  378,  379] ;  to  biological 
complexity  [203,  223,  345];  to  cognitive  models  [397];  economic  models  [452]  and  data 
compression  [499],  Lempel-Ziv  encoding  can  be  seen  as  a  computable  approximation  to  it  [279, 
495] .  Given  that  it  is  better  characterised  as  an  information  measure  rather  than  complexity,  it 
has  very  close  connections  with  entropy,  as  explored  in  [169,  420,  496,  497],  It  is  generalised  in 
[80] .  Good  summaries  of  the  many  formal  results  and  applications  can  be  found  in  [40,  41,  70, 
118,  286] .  Other  formal  results  include  [80,  284,  456,  451,  494],  A  summary  of  philosophical 
applications  can  be  found  in  [285] ,  with  others  in  [102,  301,  469] . 

(Sommerer  and  Mignonneau,  2002):  The  best-known  definition  of  complexity  is  the 
Kolmogorov-Chaitin-Solomonoff  (KCS)  definition  [62],  describing  Algorithmic  Information 
Complexity  (AIC),  which  places  complexity  somewhere  between  order  and  randomness;  that  is, 
complexity  increases  as  Pmin,  the  shortest  algorithm  that  can  generate  a  digit  sequence,  S, 
increases  to  the  length  equal  to  the  sequence  to  be  computed;  when  the  algorithm  reaches  this 
incompressibility  limit  the  sequence  is  de.ned  as  random.  The  KCS  de.nition  distinguishes 
between  “highly  ordered’’  and  “highly  complex”  structures. 

2.4  Arithmetic  Complexity 

(Edmonds,  1999):  This  is  the  minimum  number  of  arithmetic  operations  needed  to  complete  a 
task.  This  is  important  in  order  to  make  computational  algorithms  more  efficient,  for  example 
Strassen  [424]  improved  upon  Gauss's  method  for  solving  linear  equations  from  operations  to  C 
x  n2'71.  This  is  more  of  a  practical  definition  and  not  intended  as  a  general  model  of  complexity. 
The  operations  of  arithmetic  are  very >  particular.  It  also  does  not  take  into  account  the  precision 
of  the  operations  or  of  rounding  errors.  A  summary  of  the  theory >  of  the  arithmetic  hierarchy  can 
be  found  in  [178] . 

2.5  Bennett's  ‘Logical  Depth’ 

(Edmonds,  1999):  Bennett  [54,  55,  56]  defines  ‘logical  depth’  as  the  running-time  to  generate 
the  object  in  question  by  a  near-incompressible  program.  Strictly  the  depth  of  a  string  x  at  level  s 
is:  Ds  (x)  =min{T(p)p-p*  <saU(p)=x},  where  p  ranges  over  programs,  T(p)  is  the  time  taken  by 
program  p,  p*  is  the  smallest  such  program  and  U  is  a  universal  Turing  computer.  He  states  that 
this  is  intended  as  a  measure  of  the  value  of  information.  For  example,  tide-tables  can  have  a 
greater  value  than  the  equations  which  were  used  to  calculate  them  as  a  lot  of  useful  computation 
has  been  done.  Thus  he  says  [57] :  “Logically  deep  objects...  contain  internal  evidence  of  having 
been  the  result  of  a  long  computation  or  slow-to-simulate  dynamically  process  and  could  not 
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plausibly  have  originated  otherwise.  ”.  The  plausibility  of  its  origin  comes  from  the  assumption 
that  the  most  likely  program  to  produce  an  output  would  be  the  shortest  one.  This  idea  comes 
from  Solomonoff  He  justifies  this  as  a  physically  plausible  measure  of  complexity  by  its 
obedience  to  the  “slow  growth  law  ”  of  complexity.  This  informal  law  states  that  complexity  can 
only  arise  slowly  through  stochastic  processes,  as  presumably  has  occurred  in  evolution.  By  its 
construction  one  cannot  produce  a  deep  object  from  a  shallow  one  by  a  deterministic  process  and 
only  improbably  by  a  stochastic  one.  Thus  random  strings  and  very’  simple  ones  both  have  a  low 
logical  depth.  A  random  string  is  incompressible  and  hence  the  minimal  program  that  produces 
it,  is  a  simple  copying  program,  which  is  quick.  A  simple  pattern  can  be  produced  by  a  simple 
program,  and  so  will  also  be  fairly  quick.  Koppel  [268]  shows  that  Logical  Depth  is  the  same  as 
Sophistication  (section  8.42  on  page  159)  for  infinite  strings. 

2.6  Cognitive  Complexity 

(Edmonds,  1999):  In  cognitive  psychology >,  several  types  of  complexity >  are  distinguished.  The 
most  discussed  of  these  is  Cognitive  Complexity.  This  was  defined  by  Kelly  as  a  part  of  his  theory 
of  personality’  [250] .  He  developed  his  ‘role  construct  repertory’  test  to  test  it.  Since  then  it  has 
been  used  as  a  basis  for  discussion  on  the  complexity >  of  personal  constructions  of  the  real  world 
(and  particularly  of  other  people)  in  psychology >.  It  asks  the  subjects  to  rate  a  number  of  people 
known  to  them  (e.g.  closest  friend  of  same  sex)  on  a  number  of  attributes  (like  Outgoing  vs.  Shy). 
The  dimension  of  the  inferred  mental  model  of  these  people  is  then  estimated  as  their  cognitive 
complexity*.  So,  for  example,  people  who  assign  to  all  their  friends  positive  attributes  and  to  their 
enemies  negative  attributes  would  have  a  one-dimensional  mental  model  of  their  acquaintances, 
as  everybody  is  aligned  along  this  good/friend  -  bad/enemy  scale.  Such  people  are  said  to  be 
“cognitively  simple’’.  A  person  who  indicated  that  some  of  both  their  friends  and  enemies  were 
good  and  bad  would  have  at  least  a  two-dimensional  model  with  people  placed  across  a  good- 
bad,  friend-enemy  pair  of  axes.  This  person  would  have  a  higher  score  and  would  be  called 
“more  cognitively  complex”.  Thus  the  level  of  cognitive  complexity >  indicates  the  number  of 
potential  relationships  between  the  various  attributes.  Quite  a  number  of  variations  of  this  has 
been  suggested  to  capture  this  idea  [404] .  Unfortunately  these  seem  to  measure  slightly  different 
things  as  they  do  not  correlate  in  practice  [204] ,  although  they  do  have  some  robustness  over 
time  [342] .  There  does  not  seem  to  be  any  strong  connection  between  cognitive  complexity  and 
IQ  [95] ,  innovation  [187],  intellectual  sophistication  [416] ,  loquacity  [81]  or  educational  level 
[366] .  It  does  seem  to  have  some  relation  to  the  ability  to  use  complex  language  [48,  409],  The 
application  of  hierarchically  structured  algorithmic  information  is  discussed  in  [397] .  A  synthesis 
of  several  measures  of  cognitive  complexity  is  suggested  in  [413]  in  the  internal  representation 
used  by  subjects.  Other  related  measures  include:  [59,  382,  426], 

(Xing  and  Manning,  2005):  Another  line  of  complexity  studies  involves  cognitive  complexity. 
While  complexity  studies  generated  by  information  theory >  focus  on  the  complexity  of  a  system 
itself,  studies  of  cognitive  complexity  focus  on  observers:  complexity >  from  the  perspective  of  the 
observer,  i.e.,  the  users.  Since  air  traffic  control  involves  cognitive  tasks  such  as  monitoring  the 
situation,  resolving  conflicts,  issuing  instructions,  etc.,  it  is  important  to  understand  how  cog¬ 
nitive  complexity >  is  measured  to  assess  the  complexity  of  ATC  displays.  Cognition  may  best  be 
thought  of  a  construct  system  composed  of  constructs  and  elements  (Kelly  1955).  The  constructs 
are  transparent  templates  that  a  person  uses  to  comprehend  the  world.  In  a  sense  then,  humans 
create  the  templates  and  fit  the  perception  of  the  world  to  them. 
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2.7  Computational  Complexity 


(Sommerer  and  Mignonneau,  2002):  Computational  complexity  is  now  a  much  studied  area 
with  many  formal  results  [66],  The  foundation  of  complexity  theory  is  the  research  into 
computability  theory >  undertaken  since  the  1930s  onward  by  Alan  Turing,  Alonzo  Church  and 
Stephen  Kleene  [67] ,  among  others.  The  primary  considerations  then  were  the  formalization  of 
the  notion  of  a  computer  (e.g.  the  Turing  machine,  Church ’s  lambda  calculus)  and  whether  such 
a  computer  could  solve  any  mathematical  problem. 

2.8  Connectivity 

(Edmonds,  1999):  The  greater  the  extent  of  inter-connections  between  components  of  a  system, 
the  more  difficult  it  is  to  decompose  the  system  without  changing  its  behaviour.  Thus  the 
connectance  of  a  system  (especially  when  analysed  as  a  graph  [367])  becomes  a  good  indication 
of  the  potential  for  complex  behaviour,  in  particular  the  likelihood  that  the  system  will  achieve  an 
equilibrium.  The  connectivity  of  a  system  has  been  variously  measured,  including  the  number  of 
relations  (section  8.30  on  page  153)  and  the  cyclomatic  number  (section  8.7  on  page  140). 
Applications  include:  the  reliability  of  circuits  [470] ;  the  stability  of  random  linear  systems  of 
equations  [25] ;  stability  in  computational  communities  [259] ;  stability  in  ecosystems  [86,  227, 
353]:  the  diversity  of  ecosystems  [308];  the  structure  of  memory >  [273];  logical  and 
computational  properties  of  bounded  graphs  [319] ;  competition  in  networks  [373] ;  random 
digraphs  [405];  chemical  reaction  mechanisms  [491];  and  general  emergent  behaviour  in 
biological  systems  [197] . 

2.9  Crutchfield’s  Topological  Complexity 

(Sommerer  and  Mignonneau,  2002):  The  topological  complexity  described  by  James  Crutch, 
eld  [65]  is  a  measure  of  the  size  of  the  minimal  computational  model  (typically  a  .  nite  automaton 
of  some  variety >)  in  the  minimal  formal  language  in  which  it  has  a  .  nite  model.  Thus  the 
complexity >  of  the  model  is  “objectivized”  not  only  by  considering  minimal  models  but  also  as 
related  to  the  .  xed  hierarchy  of  formal  languages. 

2.10  Cyclomatic  Complexity. 

(Xing  and  Manning,  2005):  McCabe  (1976)  defined  cyclomatic  complexity  as  the  difference 
between  the  total  number  of  transitions  and  the  total  number  of  states. 

2.11  Cyclomatic  Number 

(Edmonds,  1999):  The  most  basic  graph  measure  (apart  from  the  number  of  vertices)  is  the 
cyclomatic  number  of  the  graph.  This  is  basically  the  number  of  independent  loops  in  a  graph.  It 
is  easily  calculated  by  the  formula  n(G)  =  m  -  n  +  p,  where  m  is  the  number  of  arcs,  n  the  number 
of  vertices  and  p  the  number  of  disjoint  partitions  the  graph  divides  into.  This  intuitively  captures 
the  inter-connectedness  of  a  graph  or  system;  a  hierarchically  structured  machine  is  completely 
predictable  (a  tree  has  no  loops),  whilst  one  with  many  feedback  loops  can  exhibit  more  complex 
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behaviour.  An  army  is  organised  on  hierarchical  lines,  presumably  to  simplify  the  chain  of 
command  to  make  it  more  predictable  and  hence  more  controllable.  On  the  other  hand,  a  creative 
committee  meets  to  allow  the  maximum  number  of  communication  channels  to  enable  the 
unpredictable  to  occur.  In  general  there  is  no  direct  relation  between  the  size  (number  of  nodes) 
and  the  (cyclomatic)  complexity.  If  a  system  is  represented  by  a  graph  with  the  presence  of  some 
relation  indicated  by  an  arc,  then  the  number  of  nodes  will  limit  the  cyclomatic  complexity.  This 
effect  is  only  significant  with  very:  few  nodes  as  the  number  of  possible  arcs  goes  up  exponentially 
with  the  number  of  nodes.  For  the  theory >  of  this  area  see  [436]  McCabe  [313]  uses  this  as  a 
measure  of  program  complexity,  in  particular  to  calculate  the  number  of  different  logical  paths 
through  a  program  to  gauge  how  many  tests  it  might  need.  Other  applications  include: 
complexity >  of  simulation  models  [403] ;  and  the  difficulty  of  software  maintenance  [49,  125,  232] . 
For  discussion  on  this  see  section  5.4  on  page  106. 

2.12  Descriptive/Interpretative  Complexity 

(Edmonds,  1999):  Ldfgren  [293]  writing  from  a  biological  and  psychological  context, 
distinguishes  between  descriptive  and  interpretative  complexities.  In  a  system  with  a  description 
(like  DNA)  and  its  realisation  (the  proteins  in  the  cell),  he  associates  his  two  measures  of 
complexity  with  the  two  processes  of  interpretation  and  description.  That  is  the  complexity  of 
encoding  the  realisation  into  a  descriptive  code  and  decoding  it  back  into  a  realisation  of  that 
code.  Ldfgren  chooses  Kolmogorov  complexity >  (section  8.2  on  page  136)  for  the  process  of 
description  and  an  ordering  based  on  logical  strength  (section  8.20  on  page  146)  for  the 
interpretative  complexity. 

(Sommerer  and  Mignonneau,  2002):  In  1969,  Ronald  Fagin  decided  to  study  spectra  (the 
spectrum  of  a  first-order  sentence  is  the  set  of  cardinalities  of  its  finite  models)  and  Asser’s 
problem  (1955):  “Is  the  class  of  spectra  closed  under  complementation?”  [68]  In  1970,  his 
investigations  expanded  to  generalized  spectra  (i.e.  existential  second-order  spectra  where  not  all 
relation  symbols  are  quantified  out).  Fagin ’s  most  important  result  was  probably  his 
characterization  of  NP  as  the  class  of  generalized  spectra  in  1974.  Interest  in  the  subject  has  now 
exploded,  mainly  due  to  the  intimate  relationship  (first  hinted  at  by  Fagin)  between  finite  model 
theory >  and  complexity >  theory >  [69],  In  fact,  there  is  an  established  subject  area  within  finite  model 
theory  dealing  explicitly  with  this  relationship:  descriptive  complexity  theory. 

2.13  Dimension  of  Attractor 

(Edmonds,  1999):  Chaotic  processes  are  difficult  to  model.  A  small  change  in  state  now  causes  a 
large  change  later  which  makes  it  impossible  to  predict  the  exact  state  beyond  a  certain  time 
limit.  This  does  not  mean  that  all  aspects  of  the  process  are  impossible  to  model.  It  is  possible  to 
estimate  the  processes'  attractor  in  state  space;  this  is  often  fractal  with  chaotic  processes.  The 
dimension  of  the  attractor  is  a  measure  of  how  complex  the  process  is.  Depending  on  the  method 
of  convergence  for  the  calculation  of  a  dimension  for  the  attractor  you  get  a  slightly  different 
measure.  These,  in  fact,  form  a  sequence  of  dimensions.  For  an  accessible  introduction  see  Baker 
[39]. 
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2.14  Ease  of  Decomposition 


(Edmonds,  1999):  The  ease  with  which  a  system  can  be  decomposed  into  sub-systems  has  close 
connections  with  the  “analytic  complexity’’  of  section  5.2  on  page  87.  The  general  area  is 
covered  by  [18,  110,  144,  338,  421J.  Some  techniques  for  systematic  decomposition  are:  the  use 
of  a  matrix  algorithm  to  plan  the  use  of  multiplexers  for  circuits  [278] ;  a  graphical  approach  in 
[242] ;  decomposing  difference  equations  [30,  306] ;  a  hierarchical  holographic  algorithm  [72] ; 
the  design  of  decision  support  facilities  [225] ;  a  systems  approach  [157] ;  and  a  technique  based 
on  whether  data  relations  commute  [131].  The  converse  of  and  complement  to  decomposability  is 
reconstructability  analysis  [94] . 

2.15  Economic  Complexity 

(Edmonds,  1999):  “Complexity”  in  economics,  frequently  means  merely  that  some  of  the  usual 
simplifying  assumptions  do  not  hold.  An  example  of  these  assumptions  is  that  an  agent  acts  as  if  it 
can  infer  the  action  to  perfectly  optimise  its  utility.  This  goes  back  to  Simon  ’s  distinction  between 
procedural  and  substantive  rationality  [415],  See  the  paper  in  Appendix  7  -  Complexity  and 
Economics,  for  a  full  discussion  of  the  concept  of  complexity  in  economics.  Some  papers  that 
cover  this  are  [10,  14,  20,  172,  173,  200],  In  game-theory,  there  has  been  some  more  direct 
formulation  of  actual  complexity  measures,  including:  a  critique  of  the  “number  of  states” 
measure  [43]  (section  8.32  on  page  153);  the  information  of  strategies  [290] ;  a  survey  of  the 
area  [229] .  Another  area  deals  with  choice  processes,  including:  the  group-theoretic  complexity 
of  decision  rules  [163] ;  a  survey  of  choice  processes  and  complexity  [192] ,  the  computability  of 
choice  functions  [251] ;  hierarchies  [484] ;  and  the  cardinality  of  collections  of  decisions  [52] . 

2.16  Effective  Measure  Complexity 

(Sommerer  and  Mignonneau,  2002):  Peter  Grassberger  [75]  defines  the  Effective  Measure 
Complexity  (EMC)  of  a  pattern  as  the  asymptotic  behavior  of  the  amount  of  information  required 
to  predict  the  next  symbol  to  the  level  of  granularity.  EMC  can  be  seen  as  the  dif.  cully  of 
predicting  the  future  values  of  a  stationary  series,  as  measured  by  the  size  of  regular  expression 
of  the  required  model.  A  similar  approach  is  taken  by  Badii  and  Politi. 

2.17  Entropy 

(Edmonds,  1999):  In  physics,  entropy  measures  the  level  of  disorder  in  a  thermodynamic  system. 
The  more  disordered  it  is,  the  more  information  is  needed  to  describe  it  precisely.  In  particular 
systems  with  very >  low  entropy  are  simple  to  describe  (they  don't  move  around  a  lot).  Thus 
complexity  and  entropy  can  be  associated,  although  this  was  not  intended  by  its  originators 
[408],  Entropy  based  measures  are  essentially  probabilistic.  The  Boltzman-Gibbs-Shannon 
entropy  is  most  frequently  used  in  physics,  but  Algorithmic  Complexity  can  also  be  used  if  the 
complexity  of  the  whole  ensemble  is  low  [497],  The  principle  of  maximum  entropy  [282]  has  been 
used  to  help  formalise  complexity  [114,  156,  171] .  Entropy  based  measures  have  often  been  used 
as  measures  of  complexity  including:  the  regularity  in  noisy  time  series  [354] ;  the  topology >  of 
chemical  reactions  [492] ;  coalitions  of  economic  agents  [452] ;  physical  computation  [496]:  the 
difficulty  of  system  diagnosis  [183] :  artificial  life  [371] ;  and  the  complexity  of  graphs  [332] . 
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2.18  Goodman's  Complexity 


(Edmonds,  1999):  Goodman  [186]  has  devised  an  elaborate  categorisation  of  extra-logical 
predicates,  based  on  expressiveness.  For  example,  a  general  predicate  is  deemed  more  complex 
than  a  symmetric  one,  as  it  includes  the  later  as  a  specific  example.  Likewise  a  three  place 
predicate  is  more  complex  than  a  two  place  one.  Goodman  builds  upon  this  starting  point.  The 
idea  is  that  when  faced  with  two  theories  that  have  equal  supporting  experimental  evidence  one 
should  choose  the  simpler  one  using  this  measure.  The  complexity  of  a  complex  statement  is 
merely  the  sum  of  the  complexities  of  its  component  predicates,  regardless  of  the  structure  of  the 
statement.  It  is  similar  in  spirit  to  Kemeny ’s  measure  (section  8.18  on  page  146).  A  recent  defence 
and  reformulation  of  this  idea  has  been  made  by  Richmond  in  [376] . 

(Sommerer  and  Mignonneau,  2002):  Nelson  Goodman  [71]  has  devised  an  elaborate 
categorization  of  extra-logical  predicates  based  on  expressiveness.  For  example,  a  general 
predicate  is  deemed  more  complex  than  a  symmetric  one,  as  it  includes  the  later  as  a  sped,  c 
example.  Likewise,  a  three-place  predicate  is  more  complex  than  a  two-place  one.  Goodman 
builds  upon  this  starting  point.  The  idea  is  that,  when  faced  with  two  theories  that  have  equal 
supporting  experimental  evidence,  one  should  choose  the  simpler  one  using  this  measure.  The 
complexity  of  a  complex  statement  is  merely  the  sum  of  the  complexities  of  its  component 
predicates,  regardless  of  the  structure  of  the  statement. 

2.19  Hieratical  complexity. 

(Xing  and  Manning,  2005):  This  definition  is  also  concerned  with  structural  rules.  A  complex 
system  is  often  constructed  hierarchically.  That  is,  it  is  composed  of  structures  on  several  scales 
or  levels.  These  may  be  scales  of  space  or  time,  or  levels  within  a  domain-specific  functional 
space.  For  example,  an  ATC  display  may  be  composed  of  several  windows,  consisting  of  different 
types  of  text  and  graphical  regions,  and  each  text  region  (such  as  a  datablock)  containing  several 
types  of  information.  With  this  in  mind,  Bates  and  Shepard  (1993)  assumed  that  a  system  is 
composed  of  elementary  units  with  local  structures  and  the  interconnections  between  the  local 
structures  are  governed  by  rules.  They  suggested  that  complexity  is  manifested  as  variability  in 
the  convergence  and  divergence  of  interconnections.  Then  the  dimensionality  of  local  structures, 
number  of  local  structures,  and  the  range  of  connections  all  contribute  to  the  global  complexity. 
Moreover,  if  local  regions  possess  certain  computational  abilities,  then  multiple  regions  can 
interact  to  achieve  greater  complexity. 

2.20  Hinegardner  and  Engelberg’s  Number  of  Parts  Definition. 

(Sommerer  and  Mignonneau,  2002):  Perhaps  the  simplest  measure  of  complexity  is  that- 
suggested  by  R.  Hinegardner  and  H.  Engelberg  [63] :  the  number  of  different  parts.  Hinegardner 
and  Engelberg’s  measure  evokes  “exploded”  diagrams  of  pieces  of  machinery >.  They  give  some 
indication  of  complexity,  but  leave  out  what  is  perhaps  most  important:  “organization  ”  and 
“levels  of  organization  ”  [64] . 
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2.21  Horn  Complexity 


(Edmonds,  1999):  The  Horn  complexity  of  a  propositional  function  is  the  minimum  length  of  a 
Horn  formula  (in  its  working  variables)  that  defines  that  function.  This  was  defined  by  Aanderaa 
and  Borger  [1]  as  a  measure  of  the  logical  complexity  of  Boolean  functions.  It  is  polynomially 
related  to  the  network  complexity  [2] ,  described  below  (section  8.26  on  page  151). 

(Sommerer  and  Mignonneau,  2002):  The  Horn  complexity  of  a  propositional  function  is  the 
minimum  length  of  a  Horn  formula  (in  its  working  variables)  that  de.  nes  that  function.  This  was 
de.ned  by  S.O.  Aaderaa  and  Egon  Borger  [73]  as  a  measure  of  the  logical  complexity  of  Boolean 
functions.  It  is  polynomially  related  to  network  or  circuit  complexity,  which  is  the  minimum 
number  of  logical  gates  needed  to  implement  a  logical  function  [74], 

2.22  Image  complexity 

(Xing  and  Manning,  2005):  A  digital  image  is  numerically  specified;  thus,  the  information 
content  can  be  easily  computed  using  information  theory .  Many  algorithms  have  been  developed 
to  compute  image  complexity.  The  standard  Boltzmann-Gibbs  entropy  measure  defines 
complexity  with  respect  to  a  given  size  of  a  window  of  view.  According  to  the  definition,  image 
complexity,  measured  as  configurational  entropy,  is  a  function  of  the  total  number  of 
distinguishable  spatial  arrangements  within  view  windows  of  a  given  size.  The  statistical 
paradigms  based  on  this  measure  have  shown  great  success  in  quantifying  image  complexity. 
However,  experiments  have  shown  that  information  complexity  computed  in  term  of  entropy  does 
not  correspond  to  perceived  complexity.  While  entropy  is  a  measure  of  image  disorder  and 
reflects  the  lack  of  spatial  homogeneity,  complexity  is  a  combination  of  order  and  disorder. 
Indeed,  Grassberger  (1986,  1991)  has  shown  that  complexity  is  sometimes  posited  as  a  mid-point 
between  order  and  disorder. 


2.23  Information  complexity 

(Edmonds,  1999):  The  amount  of  information  a  system  encodes  or  the  amount  of  information 
needed  to  describe  a  system  has  a  loose  connection  with  its  complexity.  As  noted  above,  there  is  a 
close  connection  between  the  amount  of  information  and  disorder.  Using  the  Algorithmic 
Complexity  (section  8.2  on  page  136)  measure  of  information,  disordered  patterns  hold  the  most 
information,  patterns  encoding  the  maximum  amount  of  information  are  indistinguishable  from 
random  patterns.  Information  can  be  measured  deterministically  using  algorithmic  information 
complexity  (section  8.2  on  page  136)  or  probabilistically  using  entropy  (section  8.12  on  page 
143).  Either  of  these  can  be  used  to  define  mutual  information  (section  8.25  on  page  151).  See 
also  section  immediately  below  (section  8.16  on  page  145).  Klir  exhibits  an  axiomatic  framework 
for  complexity  similar  to  those  I  list  in  section  5.2  on  page  87,  combined  with  the  requirement 
that  complexity  should  be  proportional  to  the  information  required  to  resolve  any  uncertainty 
[262,  263,  264,  265].  This  may  be  seen  as  a  formulation  of  Waxman’s  “problem  complexity” 
[463] .  A  number  of  approaches  which  seek  to  combine  elements  of  both  algorithmic  and  shannon 
information  include:  [170,  378,  452,  496].  Computational  complexity  has  been  extended  to  cover 
information  flow  by  adding  a  cost  function  to  the  information  used  by  a  computation  [440,  441, 
442],  Applications  include:  charting  the  increase  in  information  in  the  evolution  of  finite 
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automata  [26,  27];  the  fluctuation  of  information  in  1-D  automata  [47];  its  connection  with 
logical  depth  in  evolution  [53];  its  connections  to  computational  complexity  [138];  the 
connection  between  various  measures  of  information  via  random  vectors  [144] ;  the  regularity  of 
short  noisy  chaotic  series  [160] ;  error-prone  sections  of  programs  by  potential  information  flow 
[214,  277] ;  systems  problem  solving  [262] ;  the  classification  of  strategies  in  repeated  games 
[290] ;  the  estimation  of  the  information  of  a  pattern  [340] ;  and  a  principle  of  the  minimum 
increase  in  evolution  [399], 

(Xing  and  Manning,  2005):  Complexity  has  been  extensively  studied  within  the  field  of 
information  theory,  where  the  term  “information  complexity  (1C)  ”  is  frequently  used  to  describe 
complexity  from  the  perspective  of  a  system.  There  have  been  many  attempts  to  quantify  1C 
theoretically.  Below  we  list  some  widely  used  complexity  measures.  These  measures  do  not 
necessarily  exclude  each  other.  Instead,  they  emphasize  different  aspects  of  complexity  and  are 
somewhat  complementary. 

•  Kolmogorov  complexity. 

•  Topological  complexity. 

•  Mutual  information. 

•  Logical  depth. 

•  Kauffman ’s  complexity. 

•  Hiearchical  complexity. 

2.24  Information  Gain  in  Hierarchically  Approximation  and 
Scaling 

(Edmonds,  1999):  Attention  in  physics  has  focused  on  the  complexity  of  chaotic  physical 
processes  with  a  fractal  nature,  where  one  gets  different  behaviours  at  different  levels  of 
granularity.  In  1986  Grassberger  introduced  “Effective  Measure  Complexity”  [193],  which 
measured  the  asymptotic  increase  in  information  with  increased  scale.  He  develops  this  in  [194, 
195] .  Badii,  Politi  and  others  [33,  34,  35,  36,  130]  use  trees  of  increasingly  detailed  Markov 
models  to  approximate  a  growing  pattern.  Each  branch  off  a  node  is  a  possible  extension  of  the 
pattern  that  may  follow.  He  then  defines  the  complexity  of  the  pattern  as  the  (Shannon) 
information  gain  in  each  level  over  the  size  of  the  tree  at  the  level,  taking  the  limit  at  infinitely 
many  levels.  Any  Markov  process  has  zero  complexity.  This  is  to  reflect  the  difficulty  in  predicting 
complex  systems.  The  class  of  easily  predicted  systems  that  Badii  focuses  on  are  those  which 
exhibit  different  behaviours  at  different  levels  of  detail.  He  says  “A  system  is  complex  if  it  reveals 
different  laws  (interactions)  at  different  resolution  (coarse  grinning)  levels.  ”.  [202]  argues  that 
these  measures  assume  that  the  process  is  stationary,  i.e.  is  basically  a  Markov  process  and  [457] 
classifies  them  according  to  whether  they  are  based  on  homogeneous  or  generated  partitions  and 
whether  they  are  based  on  dynamic  or  structural  elements.  Other  papers  in  this  area  include:  [3, 
7,  28,  29,  350,  480,  490],  A  good  review  of  this  whole  area  is  [36], 
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2.25  Irreducibility 


(Edmonds,  1999):  Holists  often  use  the  word  “complexity” for  that  which  is  irreducible  [339]  (at 
least  by  current  practice).  This  is,  in  a  sense,  an  extreme  case  of  the  difficulty  of  decomposition 
(section  8.10  on  page  142).  Such  approaches  include:  [13]  where  the  importance  of  size  to 
qualitative  behaviour  is  pointed  out;  [468]  which  argues  that  the  evolution  of  multiple  and 
overlapping  functions  will  limit  reduction  in  biology >;  [11]  which  discusses  the  application  to 
public  policy  in  forestry;  [247]  which  charts  how  chaos  challenges  the  reductionist  approach; 
[257]  which  applies  this  to  modelling  organisations;  as  a  result  of  self-organisation  [205] ;  the 
incompatibility >  of  information  and  computation  [243] ;  as  a  result  of  the  epistemic  cut  between 
syntax  and  semantics  [347] ;  number  of  elements  an  instance  of  a  pattern  must  consist  of  to 
exhibit  all  the  characteristics  of  a  class  [210];  and  [323]  which  discusses  Rosen’s  approach 
[384,  389]  and  relates  this  to  the  “sciences  of  complexity” .  This  approach  to  complexity  seems 
particular  to  biology >,  for  general  surveys  of  the  connection  of  complexity  with  holism  see  [244, 
348,  388,  481] .  Some  suggest  that  this  may  be  due  to  using  the  wrong  formal  language  for 
modelling,  including  [74,  167,  246,  244,  330,  344,  410,  482,  488], 

2.26  Kauffman’s  complexity. 

(Xing  and  Manning,  2005):  Kauffman  (1993)  defined  complexity  as  the  “number  of  conflicting 
constraints.  ”  The  definition  represents  the  difficulty  of  specifying  a  successful  task  within  the 
constraints  or  “rules”  imposed.  For  example,  an  airspace  can  be  made  less  complex  by  removing 
air  traffic  constraints  such  as  military  zones,  bad  weather,  etc.  Note,  however,  that  the  definition 
is  only  concerned  with  the  complexity >  factor  of  structural  rules. 

2.27  Kemeny's  Complexity 

(Edmonds,  1999):  In  the  field  of  “ simplicity  ”,  Kemeny  [254]  attributes  an  integral  measure  of 
complexity  to  types  of  extra-logical  predicates.  He  does  it  on  the  basis  on  the  logarithm  of  the 
number  of  non-isomorphic  finite  models  a  predicate  type  has.  On  the  basis  of  this  he  gives  extra- 
logical  predicates  a  complexity  which  could  be  used  to  decide  between  equally  supported 
theories.  This  is  similar  in  style  and  direction  to  Goodman’s  measure  in  section  8.13  on  page 
143. 

(Sommerer  and  Mignonneau,  2002):  In  the  field  of  “simplicity,  ”  John  G.  Kemeny  [72] 
attributes  an  integral  measure  of  complexity  to  types  of  extra-logical  predicates,  on  the  basis  of 
the  logarithm  of  the  number  of  non-isomorphic  .  nite  models  that  a  predicate  type  has.  On  the 
basis  of  this  he  assigns  to  extra-logical  predicates  a  measured  complexity  ranking  that  could  be 
used  to  decide  between  equally  supported  theories.  This  is  similar  in  style  and  direction  to 
Goodman ’s  measure  above. 


2.28  Kolmogorov  complexity 

(Xing  and  Manning,  2005):  According  to  information  theories,  the  most  straightforward 
definition  of  complexity  is  the  minimum  description  size.  Hence  Kolmogorov  complexity  is  defined 
as  the  minimum  possible  length  of  a  description  in  some  language  (Casti,  1979).  For  instance,  if 
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a  description  can  be  greatly  compressed  without  loss  of  meaning ,  then  it  is  considered  simpler 
than  one  that  cannot.  By  this  definition,  highly  ordered  expressions  appear  as  simple  and  random 
while  maintaining  maximal  complexity. 

2.29  Layout  Appropriateness 

(Xing  and  Manning,  2005):  Tullis’  metrics  are  task  independent.  They  are  focused  on  the 
general  appearance  of  an  interface.  Therefore,  it  is  more  useful  for  predicting  user  preference 
than  user  performance  other  than  search  time.  In  contrast,  task-sensitive  metrics  are  more  useful 
in  understanding  what  users  do  with  an  interface  and  how  to  make  the  interface  more  efficient. 
For  instance,  Sears  (1994)  proposed  a  measure,  called  Layout  Appropriateness,  to  evaluate  the 
efficiency  of  the  organization  of  objects  in  an  interface.  This  metric  first  computes  the  cost  of  a 
layout  using  the  following  formula:  Cost  =  sum  (frequency  of  transition  x  cost  of  the  transition). 

2.30  Length  of  Proof 

(Edmonds,  1999):  Simpler  theorems,  on  the  whole,  need  shorter  proofs.  On  the  other  hand 
longer  proofs  are  tedious  to  follow.  Thus  it  is  natural  to  search  for  short  proofs  (e.g.  as  in  [96]). 
One  can  arbitrarily  lengthen  almost  any  proof.  This  alone  makes  length  alone  as  a  measure  of 
complexity  unsatisfactory >.  Some  short  “elegant  ”  proofs  are  very >  complicated  and  some  careful 
long  explanatory >  proofs  easy  to  follow.  For  this  measure  to  make  any  sense  needless  length  needs 
to  be  eliminated  (see  minimum  size  measures  in  section  8.24  on  page  149).  Other  papers  touching 
on  this  include:  [180,  209], 

2.31  Logical  Complexity/Arithmetic  Hierarchy 

(Edmonds,  1999):  Mathematical  proof  theorists  classify  mathematical  objects  and  processes 
according  to  the  projective  hierarchy  (sometimes  called  the  arithmetic  hierarchy).  Girard  [178] 
surveys  this  area  thoroughly.  [107]  shows  that  the  arithmetic  complexity  of  the  problem  of 
deriving  a  word  from  a  fixed  starting  point  is  arbitrarily  more  complicated  than  the  word 
problem  itself. 

2.32  Logical  depth. 

(Xing  and  Manning,  2005):  Logical  depth  is  defined  as  the  computational  cost  (time  and 
memory >)  taken  to  calculate  the  shortest  program  that  can  reproduce  a  given  object  (Bennett, 
1990).  By  this  definition,  complexity >  is  the  difficulty  of  computation  from  a  random  starting  point 
to  the  resulting  state.  This  measure  is  aimed  at  the  complexity  of  the  process  and  not  the  results. 
That  is,  it  is  a  combination  of  both  storage  and  computational  power . 

2.33  Loop  Complexity 

(Edmonds,  1999):  The  loop  complexity >  of  a  primitive  recursive  function  is  the  iteration  depth  of 
the  primitive  recursive  register  operators  in  its  definition.  Thus  x+1  would  be  level  0,  x+y  level  1 
(as  it  can  be  defined  recursively  from  x+1),  x'y  level  2  etc.  This  can  be  used  to  define  a  hierarchy 
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of  sets  LOOP n  of functions  with  loop  complexity  not  greater  than  n  [322],  This  is  one  of  the  large 
collections  of  measures  used  to  predict  the  maintainability  of  software,  for  a  survey  of  these  see 
[498]. 

2.34  Low  Probability 

(Edmonds,  1999):  The  connection  of probability  and  complexity  is  intricate.  The  probability  of  a 
highly  ordered  complex  system  arising  by  chance  is  low,  hence  sometimes  complexity  is 
associated  with  low  probability  [136],  On  the  other  hand  if  complexity  is  conflated  with 
informational  measures  such  as  entropy  (section  8.12  on  page  143)  or  algorithmic  information 
(section  8.2  on  page  136)  then  complexity  is  associated  with  high  probability.  This  has  led  many 
to  look  for  other  measures  such  that  complex  systems  will  lie  between  order  and  disorder 
including  those  in  section  8.16  on  page  145  and  section  8.46  on  page  161.  In  the  contrary 
direction  the  philosophy  of  “simplicity”  (section  6.5  on  page  129  and  section  8.37  on  page  156) 
has  lead  to  the  identification  of  a  higher  a  priori  probability  of  the  truth  of  a  theory  with  a  lack  of 
complexity.  That  this  is  mistaken  see  the  arguments  in  section  6.5  on  page  129  and  the  paper  in 
Appendix  6  -  Complexity  and  Scientific  Modelling.  An  application  of  low  probability  to  the 
difficulty  of  system  diagnosis  is  in  [183] . 

2.35  Minimum  Number  of  Sub  Groups 

(Edmonds,  1999):  The  Krohn-Rhodes  prime  decomposition  theory >  [17,  272]  tells  us  that  we  can 
decompose  any  semi-group  in  to  a  wreath  product  of  alternating  simple  groups  and  semi-groups 
of  order  3.  There  are  decompositions  which  are  minimal  in  terms  of  the  number  of  such 
alternations,  i.e.  they  have  the  least  number  of  groups  in  their  decomposition.  The  number  of  such 
groups  is  called  the  complexity  of  the  original  group.  If  you  take  the  product  of  a  group  with 
another  group  then  the  result  will  be  more  complex,  which  accords  with  our  intuitions. 

2.36  Minimum  Size 

(Edmonds,  1999):  As  discussed  in  section  3.4.3  on  page  58,  minimum  size  overcomes  some  of  the 
inadequacies  of  mere  size  as  a  complexity  measure.  It  avoids  the  possibility  of  needless  length 
and  is  nicely  independent  of  the  particular  expression  chosen.  It  would  correspond  to  using  a 
perfectly  efficient  language,  the  occurrence  of  any  redundancy  in  a  specific  expression  was 
eliminated  by  perfect  compression.  However  the  minimum  size  of  a  particular  representation  can 
still  be  a  largely  accidental  feature  of  the  description  process.  Different  ideas  are  sometimes 
more  succinctly  expressed  in  different  languages  (national  and  formal).  For  example,  to  express  a 
conjunction  in  a  negation-implication  fragment  of  classical  propositional  logic  is  necessarily 
longer  than  that  for  implication  itself.  This  would  not  mean  that  implication  was  simpler  that 
conjunction.  Minimum  size  also  ignores  any  question  of  inter-relatedness  or  relevance.  Compare 
the  cases  of  1001  inter-related  facts  about  logic  and  1001  unrelated  general  knowledge  facts 
(presuming  this  to  be  possible).  It  is  probably  possible  to  compress  the  1001  inter-related  facts 
more  than  the  unrelated  ones  because  the  very>  fact  of  their  relatedness  indicates  a  degree  of 
redundancy.  The  minimum  size  approach  to  complexity  would  thus  attribute  a  lower  complexity 
to  the  1001  related  facts  but  few  would  say  these  were  less  complex.  What  is  true  is  that  the 
system  of  unrelated  facts  holds  more  information  but  is  far  complex  (as  a  system).  In  a  way  that  is 
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similar  to  what  occurred  for  length  measures,  some  of  these  seem  to  have  had  the  label 
“complexity”  applied  post-hoc,  so  it  is  difficult  to  judge  how  seriously  they  were  meant  as  a 
complexity  measure.  The  most  frequent  special  case  of  a  minimum  size  measure  is  Algorithmic 
Information  which  corresponds  to  a  minimal  sized  Turing  Machine  (see  section  8.2  on  page  136), 
but  other  minimum  size  approaches  are  also  used  with  other  formal  languages,  in  particular 
finite  automata  (see  section  8.32  on  page  153),  this  approach  is  criticised  in  [43]  for  the  analysis 
of  equilibria  in  repeated  games.  Crutchfield  generalises  the  minimal  size  criterion  over  the  whole 
formal  language  hierarchy,  so  that  complexity  is  the  minimal  size  in  the  “lowest”  formal 
language  for  which  this  is  finite  [121] .  He  contrasts  this  complexity  measure  with  a  version  of 
effective  measure  complexity  (section  8.16  on  page  145)  which  he  calls  “statistical  complexity” 
[123],  The  method  for  finding  such  an  expression  is  given  in  [119]  and  applied  to  the  process  of 
modelling  chaotic  process  in  [120,  121].  Minimum  size  measures  have  also  been  applied  to 
capture  the  static  complexity  of  cellular  automata  in  [472] ;  and  to  a  minimal  complexity  in 
evolution  [296] . 

2.37  Mutual  Information 

(Edmonds,  1999):  If  you  have  defined  an  entropy  like  measure  (e.g.  Shannon  Entropy  or 
Algorithmic  Complexity) ,  H(A)  and  from  that  a  joint  entropy  H(A;B)  which  is  the  entropy  of  A 
and  B  joined,  then  you  can  define.  This  can  be  interpreted  as,  i.e.  the  extent  of  the  shortening 
when  considered  together  rather  than  separately.  A  high  mutual  information  between  remote 
parts  of  a  system  can  indicate  a  closely  connected  or  self-similar  system.  The  connectiveness  in 
such  a  system  can  be  the  cause  of  its  complexity.  Bennett  [55]  points  out  that  this  arises  for 
rather  different  reason  in  equilibrium  and  non-equilibrium  systems.  In  equilibrium  situations  the 
mutual  information  comes  from  the  intervening  medium  (like  in  a  gas),  in  non-equilibrium 
systems  it  must  come  from  some  other  connection.  He  points  out  that  simple  operations  like 
duplicating  and  mixing  up  random  bits  of  DNA  generate  large  amount  of  remote  non- equilibrium 
mutual  information.  [287]  shows  that  past-future  mutual  entropy  is  not  related  to  entropy  in  a 
straight-forward  manner.  [7]  formulates  “physical  complexity”  as  the  mutual  information 
(defined  relative  to  a  Landuer-Turing  Machine)  between  a  systems  and  its  universe.  Mutual 
information  has  been  applied  to  capture  some  of  the  dynamic  complexity  of  cellular  automata  in 
[289]. 

(Xing  and  Manning,  2005):  Complexity  is  indicated  by  levels  of  mutual  information  that 
measure  the  correlation  between  information  at  sites  separated  by  time  and  space  (Langton, 
1991).  This  definition  describes  the  computational  power  requirement.  For  example,  if  each 
controller  only  needs  to  handle  aircraft  within  one ’s  sector  regardless  of  traffic  in  the  next  sector, 
the  task  would  be  less  complex  because  traffic  in  the  next  sector  is  not  relevant  to  his  or  her 
problem  space. 

2.38  Network  Complexity 

(Edmonds,  1999):  Network  or  circuit  complexity  is  the  minimum  number  of  logical  gates  needed 
to  implement  a  logical  function  [400] .  This  is  very  difficult  to  compute  in  most  cases  but  some 
upper  and  lower  limits  can  be  proved.  This  measure  depends  on  the  choice  of  logic  gates  that  you 
can  use  to  build  the  circuits  from.  This  measure  has  an  immediate  importance  for  electronic 
engineers  who  seek  to  minimise  the  expense  of  logic  gates  as  in  [278] .  This  is  polynomially 
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related  to  Horn  Complexity  (section  8.14  on  page  143).  For  surveys  of  results  in  this  field  see 
[40,  142,  400] . 

2.39  Number  of  Axioms 

(Edmonds,  1999):  Meredith  and  Lukasiewics  both  put  considerable  effort  into  finding  small 
axiom  sets  for  classical  propositional  logic.  However  it  is  far  from  obvious  that  this  is  helpful. 
The  axiom  has  no  immediately  comprehensible  meaning  and  it  makes  for  an  incredibly  tortuous 
proof  theory >.  For  more  on  this  see  section  5.6.1  on  page  113. 

2.40  Number  of  Dimensions 

(Edmonds,  1999):  In  any  model  of  a  process,  the  number  of  dimensions  it  takes  is  of  critical 
importance.  A  necessarily  high  dimensional  model  has  the  potential  for  great  complexity. 
Conversely  if  there  is  a  simple  relationship  between  dimensions  in  a  model  you  can  often  reduce 
the  models  dimension  by  forming  composite  dimensions  with  out  any  loss  of  descriptive  power. 
Hence  if  a  model  is  necessarily  of  high  dimension  then  there  is  no  very  simple  relationship 
between  any  of  its  several  dimensions,  i.e.  the  model  must  be  reasonably  complex.  This  has  been 
applied  to  concept  learning  [298];  the  performance  of  connecting  networks  [307]  and  in 
cognitive  complexity  (section  8.5  on  page  139).  Fractal  dimension  is  used  to  measure  plant 
development  in  [113] . 

2.41  Number  of  Inequivalent  Descriptions 

(Edmonds,  1999):  If  a  system  can  be  modelled  in  many  different  and  irreconcilable  ways,  then 
we  will  always  have  to  settle  for  an  incomplete  model  of  that  system.  In  such  circumstances  the 
system  may  well  exhibit  behaviour  that  would  only  be  predicted  by  another  model.  Thus  such 
systems  are,  in  a  fundamental  way,  irreducible.  Thus  the  presence  of  multiple  inequivalent  models 
are  considered  by  some  as  the  key  characteristic  of  “complexity’”.  These  people  are  usually 
holists,  namely  [323,  389].  See  also  section  4.3.1  on  page  83.  This  approach  can  be  extended  in 
restricted  circumstances  to  measuring  complexity >  by  the  number  of  inequivalent  descriptions  [88, 
89]. 

(Sommerer  and  Mignonneau,  2002):  If  a  system  can  be  modeled  in  many  different  and 
irreconcilable  ways,  then  we  will  always  have  to  settle  for  an  incomplete  model  of  that  system.  In 
such  circumstances,  the  system  may  well  exhibit  behavior  that  would  only  be  predicted  by  another 
model.  Thus  such  systems  are  fundamentally  irreducible.  Accordingly,  the  presence  of  multiple 
inequivalent  models  has  been  considered  by  Robert  Rosen  [76]  and  Howard  Pattee  [77]  as  the 
key  characteristic  of  complexity.  Casti  [78]  extends  this  approach  and  de.nes  complexity  as  the 
number  of  nonequivalent  descriptions  that  an  observer  can  generate  for  a  system  with  which  he 
or  she  interacts.  The  observer  must  choose  a  family  of  descriptions  of  the  system  and  an 
equivalence  relation  among  them —  the  complexity  is  then  the  number  of  equivalence  classes  the 
family  breaks  down  into,  given  the  equivalence  relation. 
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2.42  Number  of  Internal  Relations 


(Edmonds,  1999):  If  one  is  focusing  on  the  topology >  of  a  model,  then  one  improvement  on  the 
simple  size  of  the  network  as  an  indication  of  its  complexity >  is  the  number  of  relations  indicated 
between  the  nodes.  Rouse  and  Rouse  [392]  in  their  study  of  the  time  taken  to  complete  tasks 
found  a  strong  correlation  behveen  the  time  taken  to  perform  fault  diagnosis  tasks  with  complex 
relations  and  the  number  of  internal  relations  in  that  circuit  (represented  by  a  wiring 
connection).  Van  Emden  [450]  examines  the  mathematics  of  a  variety >  of  entropic  measures  based 
on  the  information  indicated  by  the  internal  relations  at  different  levels. 

2.43  Number  of  Spanning  Trees 

(Edmonds,  1999):  An  interesting  graphical  measure  is  the  number  of  spanning  trees  of  a  graph 
(see  also  section  8.7  on  page  140).  A  spanning  tree  is  a  subgraph  with  no  loops  which  includes 
all  the  vertices.  The  number  of  spanning  trees  grows  very:  fast  with  the  cyclomatic  number  and 
size  of  the  graph.  A  tree  has  only  one  spanning  tree  [252] .  [58]  applies  this  to  a  classification  of 
games.  [226,  237]  use  such  trees  as  the  basis  for  a  measure  of  complexity  to  capture  the  variety 
in  the  structure  of  trees. 

2.44  Number  of  States  in  a  Finite  Automata 

(Edmonds,  1999):  Much  formal  work  [231]  has  been  done  on  the  number  of  states  of  finite 
automata.  In  these  works  this  number  is  frequently  taken  as  the  complexity  (e.g.  [164]).  Again  it 
is  easy  to  elaborate  a  model  by  adding  redundant  states,  a  difficulty  which  is  circumvented  by 
selecting  a  minimal  or  “acceptable”  model  (see  section  8.24  on  page  149  above).  Gaines  [165]  is 
pessimistic  about  a  useful  general  theory  of  complexity,  saying:  “The  ordering  of  models  in  terms 
of  complexity  is  arbitrary  and  depends  upon  our  individual  points  of  view.  ”,  and  again:  “When 
we  specify  an  order  relation  upon  the  models  we  may  find  that  the  behaviours  of  many  important 
systems  require  complex  models  under  our  ordering,  whereas,  with  a  different  ordering  on  the 
same  class  of  models,  they  all  become  simple.  ”.  He,  nonetheless,  introduces  the  useful  concept  of 
admissibility  (borrowed  from  statistics  [465])  and  applies  it  to  the  search  for  simple  finite-state 
automata  for  various  string  patterns.  He  uses  a  working  definition  of  complexity  which  counts  the 
number  of  states  of  an  automaton  and  then  goes  on  to  identify,  with  the  help  of  the  program 
ATOM,  admissible  models  of  various  sizes.  Here  models  are  said  to  be  admissible  if  any  other 
model  that  gives  a  better  approximation  of  the  behaviour  is  more  complex  (in  the  sense  of  number 
of  states).  In  [164]  he  shows  that  even  a  small  amount  of  randomness  can  cause  an  indefinite 
increase  in  an  induced  automata  model.  This  work  is  extended  in  [396]  to  stochastic  automata. 
Von  Neuman  speculated  that  there  was  a  critical  threshold  which  allowed  self-reproduction 
[454],  In  [303]  it  is  shown  that  Turing  machines  with  very >  few  states  can  exhibit  complex 
behaviour.  Complexity  as  the  number  of  states  in  a  finite  automata  has  been  widely  applied:  to 
characterise  the  emerging  complexity  resulting  from  the  actions  of  cellular  automata  [289,  472, 
473,  474,  476] ;  in  economic  game  theory  [229] ;  to  characterising  social  structure  [9]  and  in 
characterising  the  computation  done  in  chaotic  systems  [122] . 
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2.45  Number  of  Symbols 


(Edmonds,  1999):  The  number  of  symbols  is  not  a  reliable  guide  to  complexity.  Merely  to  count 
the  number  of  symbols  in  philosophical  works  would  give  one  little  indication  of  their  complexity. 

2.46  Number  of  Variables 

(Edmonds,  1999):  The  number  of  variables  in  a  statement  can  have  an  immediate  impact  both  on 
proofs  that  use  it  and  the  complexity  of  its  models.  Both  of  these  effects  depend  on  the  structure  of 
the  statement.  For  example  the  axiom  a->b  has  a  catastrophically  simplifying  effect  on  both 
proofs  and  models  compared  to  that  of  a->a.  As  with  the  number  of  symbols  (section  8.33  on 
page  154  above)  the  number  of  variables  can  have  a  limiting  effect  on  complexity  but  the  number 
of  variables  is  not  a  sufficient  condition  for  complexity.  Diamond  and  McKinsey  proved  [139] 
that  for  a  broad  range  of  logics  you  need  at  least  one  axiom  with  three  variables  in  it. 

2.47  Organised/Disorganised  Complexity 

(Edmonds,  1999):  Weaver  [464]  classified  scientific  problems  into  the  simple,  and  the  complex. 
Then  he  further  classified  the  complex  problems  into  those  of  disorganised  complexity  and 
organised  complexity.  Simple  problems  are  those  with  a  few  variables  like  the  path  of  a  billiard 
ball  and  a  complex  problem  is  one  with  many  variables  like  a  gas.  Disorganised  complexity  is 
typified  by  many  independent  variables,  so  that  it  is  amenable  to  statistical  techniques.  Examples 
of  this  are  the  properties  of  a  gas  or  a  nation's  accident  statistics.  Organised  complexity  occurs 
when  “There  is  a  sizeable  number  of  factors  which  are  interrelated  into  an  organic  whole” 
Examples  given  by  him  include  the  immune  system  of  animals  and  economic  fluctuations. 

2.48  Pattern  complexity 

(Xing  and  Manning,  2005):  Unlike  Boltzmann-based  complexity,  pattern  complexity  of  an  image 
is  based  on  measures  of  visual  features.  Orland  et  al.  developed  an  algorithm  to  measure  pattern 
complexity  (Orland,  Weidemann,  Larsen,  &  Radja,  1994).  Pattern  complexity  includes  measures 
of  color,  edges,  fractal  dimensions,  deviation  and  entropy.  While  the  measure  is  somewhat 
correlated  to  human  judgment  of  image  appearance,  it  is  not  a  solid  predictor  of  perceived 
complexity.  Klinger  and  Salingaros  (2000)  proposed  a  pattern  complexity  index  based  on  the 
following  visual  features:  size,  density,  line  curvature,  color,  symmetry >,  similarity  of  shapes,  and 
correctness  of  form.  In  their  algorithm,  complexity  is  composed  of  two  components:  Harmony 
and  Temperature.  Harmony  H  measures  the  correlation  of  subunits  via  symmetries;  Temperature 
T  measures  symbol  variation.  The  temperature  components  for  complex  structures  were:  1) 
intensity  and  size  of  details;  2)  differentiation  density;  3)  line  curvature;  4)  color-intensity;  and  5) 
color-contrast.  Harmony  is  a  similar  five-part  sum  composed  of  the  following  symmetry  values: 
1)  vertical  and  horizontal  reflections;  2)  translations  and  rotations;  3)  shape-similarity;  4)  form- 
connectedness;  and  5)  color-matching.  Pattern  complexity  can  then  be  computed  as  C  =  T  (Hmax 
-H). 
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2.49  Shannon  Information  &  Concept  of  Entropy 


(Edmonds,  1999):  Although  Shannon  [408]  did  not  envisage  his  measure  of  information  being 
used  to  quantify  complexity,  some  of  his  successors  have  either  used  it  as  such  or  based 
complexity  measures  upon  it.  The  Shannon  measure  of  information  is  a  statistical  measure  based 
on  the  probability  of  receiving  a  message.  If  p(ml),  p(m2),  ...  are  the  probabilities  of  receiving 
the  messages  ml,  m2,  ...  then  the  information  carried  by  the  message  nl,  n2,  ...  is  defined  as  - 
SUMi(log2(p(ni)).  The  more  improbable  the  message,  the  more  information  it  gives  the  recipient. 
See  the  section  on  information  (section  8.15  on  page  144)  and  entropy  (section  8.12  on  page 
143). 

(Sommerer  and  Mignonneau,  2002):  Shannon  Entropy  [70]  can  be  seen  as  the  dif.culty  of 
guessing  the  content  of  a  message  passing  down  a  channel,  given  the  range  of possible  messages. 
The  idea  is  that  the  more  dif.cult  it  is  to  guess,  the  more  information  a  message  holds.  This 
concept  was  not  intended  as  a  measure  of  complexity,  but  has  been  used  as  such  by  subsequent 
authors. 

2.50  Simplicity 

(Edmonds,  1999):  When  faced  with  two  theories  which  are  equally  supported  by  the  available 
experimental  evidence,  it  is  natural  to  choose  the  simpler  of  the  two.  Further  than  this,  when  a 
theory  has  been  elaborated  in  order  to  explain  the  evidence,  it  is  often  fruitful  to  search  for  a 
simpler  theory >.  The  study  of  the  grounds  for  choosing  between  equally  supported  theories  has 
acquired  the  label  “Simplicity”  (see  [4,  78,  186,  253,  254,  339,  485]).  From  the  point  of  view  of 
theories  about  the  world,  all  purely  logical  propositions  are  equally  and  ultimately  certain  and 
hence  “simple”.  Thus  measures  of  simplicity  do  not  help  us  to  distinguish  between  logical 
theories,  they  were  not  meant  to.  Many  theories  of  Simplicity  have  chosen  grounds  other  than 
simplicity  as  the  criterion  for  choosing  between  equally  supported  theories,  e.g.  Popper's 
refutability  [358]  or  Defrays  [241]  identification  of  Simplicity  with  high  probability.  Some 
theories  with  connections  with  complexity  are  Goodman's  (section  8.13  on  page  143),  Kemeny's 
(section  8.18  on  page  146)  and  Sobers  (section  8.41  on  page  158).  For  a  fuller  discussion  of  this 
see  section  6.5  on  page  129  and  Appendix  6  -  Complexity  and  Scientific  Modelling. 

2.51  Size,  Number  of  Parts 

(Edmonds,  1999):  There  is  clearly  a  sense  in  which  people  use  “ complexity  ”  to  indicate  the 
number  of parts  but  seems  rarely  used  just  to  indicate  this.  It  would  be  odd  for  a  person  opening 
a  phone  book  or  a  large  box  of  matches  to  exclaim  “Oh,  how  complex!”.  Contrast  these  examples 
with  those  of  a  mathematical  text  book  or  an  intricate  (old  fashioned)  watch,  where  this  would  be 
more  appropriate.  Size  seems  not  to  be  a  sufficient  condition  for  complexity.  On  the  other  hand  a 
certain  minimum  size  does  seem  to  be  a  necessary  condition  for  complexity.  It  is  very:  hard  to 
imagine  anything  complex  made  of  only  two  parts.  However,  this  minimum  size  can  be  quite 
small:  small  non-abelian  mathematical  groups  can  be  very >  complex  indeed  as  are  many  other 
formal  systems  with  a  sparse  axiomatisation.  The  rate  of  potential  complexity  seems  to  increase 
very  fast  with  size.  This  does  not,  of  course,  mean  that  all  large  systems  are  complex.  Size  based 
measures  of  complexity  seem  to  come  about  in  two  circumstances:  as  a  result  of  a  post-hoc 
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labelling  of  a  formal  device  (as  in  simple  induction  proofs  where  the  length  of  a  proof  the 
number  of  connectives  or  the  depth  of  nesting  is  in  need  of  a  convenient  label)  and  to  indicate  a 
potential  for  complexity  (as  in  the  number  of  variables  in  a  formula).  Anderson  points  out  that 
size  can  make  a  qualitative  difference  to  the  behaviour  of  systems  [13]  as  [454]  also  suggests,  but 
[303]  indicates  that  in  the  presence  of  powerful  inferential  machinery >  that  the  critical  size  can  be 
very:  small.  Applications  include:  the  social  organisation  and  community  size  [83] ;  the  minimum 
number  of  gates  in  a  circuit  [278]:  the  cyclical  behaviour  of  systems  [458];  self-replicating 
sequences  [44]:  rule-based  systems  [341];  neural  networks  and  cellular  automata  [188] ;  and 
grammatical  development  [256],  See  also  the  discussion  in  section  3.4.1  on  page  57  and  the  other 
size  and  numerosity  based  approaches  in  this  appendix. 

2.52  Size  of  Grammar 

(Edmonds,  1999):  A  pattern,  if  viewed  as  the  result  of  production  rules  in  a  language,  has  a 
grammar  [231] .  In  general  the  simpler  the  pattern,  the  simpler  the  grammar.  So  the  size  of  the 
grammar  gives  us  a  handle  on  the  complexity  of  the  pattern.  The  size  and  complexity  Syntactic  of 
the  grammar  can  vary  depending  on  what  sort  of  language  you  are  assuming  the  pattern  to  be  a 
representative  of.  For  instance  Gaines  [164]  shows  that  the  assumption  that  a  process  can  be 
modelled  by  a  deterministic  finite  automaton  leads  to  very >  large  models  (proportional  to  the 
length  of  the  evidence)  in  the  presence  of  even  a  small  amount  of  indeterminism.  This  measure 
would  identify  all  patterns  of  a  particular  language  as  equally  complex  unless  the  pattern 
happens  also  to  be  a  member  of  another  language  as  well.  Sahal  [396]  demonstrates  similar 
results,  but  with  stochastic  automata.  Frequently  the  size  of  grammar  is  taken  relative  to  a  Turing 
machine  (section  8.2  on  page  136)  or  finite  automata  (section  8.32  on  page  153).  Other 
approaches  include  simple  depth  (section  8.44  on  page  160)  or  star  height  in  regular  languages 
[153],  Applications  include:  biological  macromolecules  [146]:  chaotic  systems  in  physics 
(section  8.16  on  page  145);  and  communication  complexity  [235], 

2.53  Size  of  matrix 

(Edmonds,  1999):  The  size  of  a  minimal  characteristic  matrix  for  a  logic  is  an  indication  of  the 
logic's  complexity  [209] .  Classical  logic  has  the  smallest  possible  matrices  (2x2),  and  more 
complex  logics  like  R,  do  not  have  finite  characteristic  matrices  at  all.  This  measure  is  an 
indication  of  the  the  complexity  of  logic's  semantics  but  does  not  have  a  direct  relationship  with 
the  complexity  of  its  proof  theory  (see  section  5.6.2  on  page  117).  This  sort  of  approach  has  been 
applied  to:  the  stability >  of  computational  communities  [259] ;  flow  dominance  in  layout  problems 
[215] ;  and  hierarchical  decomposition  of  systems  [72], 

2.54  Sober's  Minimum  Extra  Information 

(Edmonds,  1999):  In  the  field  of  the  “simplicity:”  of  scientific  theories,  Sober  [418]  rejected  the 
idea  of  an  absolute  measure  in  favour  of  that  of  an  ordering  based  on  how  much  extra 
information  would  be  needed  to  select  an  answer  to  a  particular  question:  this  is  implicitly  as 
relativised  informativeness.  Thus  simplicity  was  to  be  relative  to  a  question  (represented  by  a  set 
of  possible  answers).  The  theory  that  needs  the  least  minimum  extra  information  to  select  an 
answer  to  the  question  is  deemed  the  simpler  one.  When  one  is  judging  theories  with  respect  to  a 
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number  of  questions  one  must  decide  a  weighting  of  the  relative  importance  of  the  questions,  to 
decide  the  overall  simplicity.  Sober  applies  this  to  mathematical  and  logical  fields  by  examining 
how  the  fundamental  axioms  are  chosen.  This  is  done  by  seeing  how  much  information  they 
contribute  to  the  question  of  whether  the  axiom  is  true  in  our  world  or  not.  According  to  Sober 
“This  mirrors  our  belief  that  a  contraction  in  the  axiom  set  is  a  gain  in  simplicity.  Moreover,  a 
proof  that  the  axioms  are  mutually  independent  is  a  proof  that  the  axiom  set  is  maximally  simple; 
no  axiom  is  redundant.  And  a  proof  that  the  axiom  set  is  complete  simplifies  our  view  of  the  area 
being  axiomatised,  for  it  assures  us  that  relative  to  the  axiom  set,  every >  truth  is  redundant.  ”  Next 
Sober  considers  some  logical  properties  of  (extra-logical)  relations  by  considering  the 
informativeness  of  them  relative  to  the  general  question  of  whether  two  objects  are  related.  Thus 
he  arrives  at  similar  conclusions  to  Goodman  (e.g.  a  symmetrical  relation  is  simpler  that  an  anti- 
symmetrical  one  etc.). 

2.55  Sophistication 

(Edmonds,  1999):  Koppel  [268]  defines  “sophistication  ”  as  a  measure  of  the  structure  of  a 
string.  He  says:  “The  minimal  description  of  a  string  consists  of  two  parts.  One  part  is  a 
description  of  the  string's  structure,  and  the  other  part  specifies  the  string  from  among  the  class 
of  strings  sharing  that  structure  (Cover  1985).  The  sophistication  of  a  string  is  the  size  of  that 
part  of  the  description  which  describes  the  strings  structure.  Thus,  for  example,  the  description  of 
the  structure  of  a  random  string  is  empty  and  thus,  though  its  complexity  is  high,  its 
sophistication  is  low.  ” 

2.56  Stochastic  Complexity 

(Edmonds,  1999):  Rissanen  [379]  finds  the  idea  of  “shortest  code  length’’  (like  algorithmic 
complexity)  attractive  but  difficult  to  apply  when  modelling  physical  processes.  He  estimates  the 
minimum  code  length  of  data  encoded  with  a  probabilistic  model,  using  Shannon's  coding  theory >. 
This  can  be  seen  as  a  statistical  and  computable  version  of  algorithmic  information  (section  8.2 
on  page  136)  as  well  as  an  attempt  to  establish  a  principled  trade-off  between  a  model’s 
complexity  and  error  rate  (see  Appendix  6  -  Complexity  and  Scientific  Modelling).  Re-christened 
as  the  minimum  description  length  (MDL)  principle  [378],  it  has  been  successfully  applied  to 
machine  learning  [377,  493] . 

2.57  Structural  Complexity 

(Xing  and  Manning,  2005):  In  simplest  terms,  absolute  structural  complexity  equals  the  number 
of  states  (Stevens,  Myers,  &  Constantine,  1974).  Relative  structural  complexity  is  the  ratio  of  the 
number  of  transitions  to  the  number  of  states,  i.e.,  the  number  of  transitions  per  state. 

2.58  Structure  density. 

(Xing  and  Manning,  2005):  Kornwachs  (1987)  proposed  “structure  density”  as  a  measure  of 
system  complexity.  This  measure  estimates  the  actual  density  of  transitions  compared  with  the 
maximal  possible  density.  Let  S  be  the  number  of  all  possible  states  of  a  system  and  T  be  the 
number  of  actual  transitions.  The  maximal  possible  number  of  transitions  is  S  *  (S-l).  Then  the 
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structure  density  is  defined  as  T/(S*(S-1 )).  By  this  definition,  the  structure  density  of  the  above 
example  is  4/(3*(3-l))=0.66. 

2.59  Syntactic  Depth 

(Edmonds,  1999):  The  deeper  phrases  are  embedded  in  a  statement  (according  to  some  syntax), 
the  more  difficult  they  are  to  understand.  Identifying  the  ease  of  comprehension  is  one  of  the 
primary  purposes  of  measures  of  syntactic  complexity  in  formal  language  theory.  In  1960  Yngve 
[483]  proposed  depth  of  postponed  symbols  as  a  measure  of  syntactic  complexity,  this  was 
criticised  by  Miller  and  Chomsky >  [326]  on  formal  grounds.  They  preferred  the  degree  of  self- 
embedding  because  it  was  “ ...  precisely  the  property  that  distinguishes  context-free  languages 
from  the  regular  languages.  ”  Other  measures  proposed  in  [383]  include  depth  and  nesting.  The 
depth  of  a  syntactic  expression  is  the  maximum  number  of  arcs  from  root  to  leaf  when 
represented  in  a  tree  form.  This  has  nothing  to  do  with  either  “logical  depth”  (section  8.4  on 
page  138)  or  “thermodynamic  depth”  (section  8.46  on  page  161).  From  the  point  of  view  of  a 
modeller,  depth  is  a  useful  way  to  stratify  a  space  of  expressions  in  a  recursive  language,  as 
typically  the  number  of  possible  expressions  goes  up  exponentially  with  the  depth.  Although  as 
[349]  points  out  this  could  be  done  in  any  number  of  ways.  Thus  depth  is  relevant  to  the  problem 
of  induction  whether  by  humans  (see  section  6.5  on  page  129  and  Appendix  6  -  Complexity  and 
Scientific  Modelling)  or  in  machine  learning  [111,  148,  333] .  Syntactic  depth  as  an  indication  of 
complexity  has  also  been  applied  to  menu  design  [239] ;  the  difficulty  of  resolution  of  ambiguity 
[174]  and  circuit  design  [355,  356], 

2.60  Tabular  Complexity 

(Edmonds,  1999):  Tabular  complexity  is  an  adaptation  of  Kolmogorov's  e-entropy  [267]  by 
Vitushkin  [453],  It  is  a  measure  of  the  complexity  of  finite-state  automata  (see  also  section  8.32 
on  page  153).  To  calculate  it  one  takes  the  tables  representing  the  change  of  state  and  the  output 
of  the  semi-group  of  the  states  of  the  automata  and  then  decomposes  these  tables  into  smaller 
sub-tables,  also  allowing  for  the  decomposition  of  the  “ wiring  ”  (the  connections)  between  these 
sub-tables  etc.  The  minimum  total  volume  obtainable  is  the  tabular  complexity,  i.e.  it  is  the 
volume  of  the  most  compact  tabular  representation.  Thus  tabular  complexity  is  similar  to  It  is 
only  applicable  to  processes  modellable  by  finite  automata  (a  proper  subset  of  those  computable 
by  a  Turing  Machine) .  The  tabular  complexity  can  be  very  difficult  to  calculate  but  estimates  can 
be  produced  by  exhibiting  specific  tables. 

2.61  Time  and  Space  Computational  Complexity 

(Edmonds,  1999)  Computational  complexity  is  now  a  much  studied  area  with  many  formal 
results.  It  is  usually  cast  as  the  order  of  the  rate  of  growth  of  the  resources  needed  to  compute 
something  compared  to  the  size  of  its  input.  Such  time  and  space  complexity  measures  are  the 
most  studied  computational  measures.  Articles  which  include  the  word  complexity  often  refer  to 
these.  They  reflect  the  degree  of  effort  required  to  compute  a  problem,  independent  of  particular 
instances  of  that  problem.  They  are  fairly  rough  measure  because  they  only  give  the  degree  of 
increase  to  within  a  constant  factor,  e.g.  the  order  of  the  polynomial  with  which  they  increase. 
This  is  because  of  possible  variations  in  the  abstract  computer  that  does  the  calculation.  Several 
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variations  of  this  have  been  proposed,  including:  extension  to  other  fields  like  the  real  numbers 
[66];  continuous  complexity  models  [318];  information  based  complexity  (which  adds  a  cost 
function  to  the  information  used)  [440,  441,  442] ;  and  using  uniform  rather  than  logarithmic  size 
[211] .  Applications  include:  social  choice  theory  [251] ;  grammatical  inference  [1 55] (learning 
[184];  simplification  in  logic  [315];  feasibility  of  reasoning  by  a  limited  agent  [281]; 
communication  [235];  induction  [111];  simulation  [335];  control  theory  [486,  487] ;  perceptrons 
[500] ;  improving  performance  on  3-SAT  problems  [230] ;  and  propagation  in  boolean  circuits 
[425] .  Summaries  of  the  field  can  be  found  in  [40,  41,  70] . 

2.62  Topological  complexity. 

(Xing  and  Manning,  2005):  Crutchfield  and  Young  (1989)  extended  the  concept  of  Kolmogorov 
complexity  by  defining  complexity  as  the  minimal  size  of  a  model  representation  of  a  system  that 
can  statistically  reproduce  the  observed  data  within  a  specified  tolerance.  Consider,  for  example, 
two  air  traffic  cases.  In  the  first  case,  ten  aircraft  are  flying  on  two  fixed  routes  that  have  one  in¬ 
tersection.  In  the  second  case,  ten  aircraft  are  flying  off  the  routes,  which  can  create  many 
potential  conflicts.  A  controller  can  build  a  model  of  the  first  case  that  has  two  flows  of  aircraft 
and  one  crossing  point,  while  a  model  of  the  second  case  has  to  be  composed  of  many  flows  and 
crossings.  Thus  the  topological  complexity  of  Case  A  is  less  than  Case  B.  This  definition  takes 
into  account  both  the  minimal  size  and  the  fixed  hierarchy  or  structural  rules  of  a  system.  One 
shortcoming  of  the  definition  is  that  it  does  not  provide  a  unique  measure  of  complexity  for  a 
system  because  there  is  not  necessarily  a  “minimal”  model  for  it  (Pressing,  1999).  That  is,  users 
may  construct  different  models  of  the  same  system.  In  addition,  neither  this  nor  the  definition 
above  is  sufficient  to  describe  complexity  because  they  only  emphasize  the  storage  resource  that 
it  takes  to  solve  a  class  of  problems. 

2.63  Tulis'  Display  complexity 

(Xing  and  Manning,  2005):  Perhaps  the  most  useful  tool  to  quantify  the  information  and  layout 
of  screen  elements  is  Tullis’  metric  of  display  complexity  (Tullis,  1984,  1985,  1986).  Tullis 
studied  over  a  thousand  computer-generated  displays.  He  measured  search  time  to  locate  items 
on  the  displays  and  collected  subjective  ratings  of  ease  of  use.  The  results  revealed  that  four  basic 
characteristics  of  display  formats  affect  how  well  users  can  extract  information  from  the  displays: 

1.  Overall  density  —  the  number  of  characters  displayed,  expressed  as  a  percentage  of  the  total 
spaces  available. 

2.  Local  density  —  the  number  of  other  characters  near  each  character. 

3.  Grouping  —  the  number  of  groups  and  average  group  size,  both  describing  the  extent  to  which 
characters  on  the  display  form  perceptual  groups.  The  groups  can  be  determined  by  considering 
the  white  space  around  them. 

4.  Layout  complexity  —  the  extent  to  which  the  arrangement  of  items  on  the  display  follows  a 
predictable  visual  scheme,  typically  computed  as  the  differences  in  view  angles  between  the  items. 


DRDC  Valcartier  TN  2006-451 


25 


2.64  Variety 


(Edmonds,  1999):  A  complex  system  is  likely  to  exhibit  a  greater  variety  in  terms  of  its  behaviour 
and  properties.  Thus  variety  is  an  indication  of  complexity  (though  not  always  as  sometimes  a 
very:  complex  system  is  necessary >  in  order  to  maintain  equilibrium).  Variety  can  be  measured  by 
the  simple  counting  of  types,  the  spread  of  numerical  values  or  the  simple  presence  of  sudden 
changes.  In  this  way  it  overlaps  with  information  (section  8.15  on  page  144)  and  entropic 
(section  8.12  on  page  143)  measures.  Applications  include:  punctuated  behaviour  [38] ;  stability 
of  ecosystems  [353] ;  competing  behaviours  and  control  [357] ;  tree  structures  [237] ;  number  of 
inequivalent  models  [89] ;  the  interaction  of  connectivity  and  complexity  [218] ;  and  evolution 
[316]. 
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3  Conclusion 


A  number  of  64  different  ways  of  formulating  complexity  are  listed  and  briefly  described  in  this 
document.  While  this  list  is  not  complete  it  provides  the  reader  a  first  reference  for  finding  most 
currently  used  key  words  on  this  subject.  By  using  proposed  searching  methods  described  in 
Chapter  1,  the  reader  will  then  be  able  to  find  the  needed  information  1)  in  this  document;  2)  in 
Edmonds  (1999);  or  3)  over  Internet. 
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